DOCUMENT RESUME 



ED 3A9 866 



HE 025 783 



AUTHOR 
TITLE 



PUB DATE 
NOTE 



PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Knight, William E. 

An Examination of Freshmen to Senior General 
Education Gains across a National Sample of 
Institutions with Different General Education 
Requirements Using a Mixed-Effect Structural Equation 
Model. AIR 1992 Annual Forum Paper. 
May 92 

26p.; Paper presented at the Annual Forum of the 
Association for Institutional Research (32nd, 
Atlanta, GA, May 10-13, 1992). 
Reports - Research/Technical (143) — 
Speeches/Conference Papers (150) 

MF01/PC02 Plus Postage. 

Academic Ability; Academic Achievement; ^College 
Outcomes Assessment; Comparative Analysis; 
Curriculum; ^General Education; Higher Education; 
Institutional Research; Interaction; Models; 
*Required Courses; Research Methodology; ^Structural 
Equation Models; ^Student Development; Undergraduate 
Study 
*AIR Forum 



ABSTRACT 

This study investigated differences in freshmen to 
senior student general education gains across institutions of higher 
education, with varying patterns of general education requirements 
using a mixed-effect structural equation model. The subjects were 
6,409 students at 34 colleges and" universities nationwide. Study 
procedures used American College Testing Assessment composite scores 
as the observed measures with the Pre-College Ability latent 
variable, and groups were based on data from students attending 
institutions with different general education requirements. Findings 
indicated that students attending institutions where less than 40 
percent of undergraduate curricular requirements were devoted to 
general education and where there was not equal distribution of 
general education courses within the requirement were found to have 
significantly higher general education gains than did students who 
attended institutions where 40 percent or more of the undergraduate 
curriculum was devoted to general education and there was equal 
distribution of courses within the general education requirement. The 
study also demonstrated the usefulness of the structural equation 
modeling for representing the interactions between student 
characteristics and general education program effects* Included are 1 
figure, 7 tables, and 23 references. (JB) 
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FRESHMEN TO SENIOR GAINS 3 
AN EXAMINATION OF FRESHMEN TO SENIOR GENERAL EDUCATION GAINS 
ACROSS A NATIONAL SAMPLE OF INSTITUTIONS WITH DIFFERENT GENERAL 
EDUCATION REQUIREMENTS USING A MIXED-EFFECT STRUCTURAL EQUATION 
MODEL 

While interest in longitudinal studies of student knowledge and skill gains in general 
education has been widespread, numerous methodological problems exist in the traditional 
study of student growth and development. Several authors (Baird, 1988; Pascarella, 1989) 
have concluded that the difficulties associated with the classic methods of analyzing change 
vitiate the benefits associated with a longitudinal or Value-added" approach. 
+ Student learning in the area of general education has been of particular interest at 
the University of Tennessee at Knoxville (UTK). A report by Pike, Phillippi, Banta, Bensey, 
Milbourne, and Columbus (1991, May) documented the results of several studies carried out 
on freshmen to senior UTK student gains on the ACT COMP Objective Test of gene al 
education knowledge and skills. Pike (1991) additionally used UTK student COMP score 
data to examine freshmen to senior gains and their relationship to patterns of course work 
through a mixed-effect structural equation modeling technique. This approach provided for 
an alternative method which was not burdened by the inadequacies of traditional methods 
of studying change as discussed by Baird (1988) and Pascarella (1989) and which allowed 
for error-free measurement of student general education learning. 

Pike (1991) listed relatively small sample size (N=704) and the reliance on a single 
outcome measure as limitations of his study. However, reliance on student ACT COMP 
scores from a single institution was another limitation. A number of studies cited most 
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FRESHMEN TO SENIOR GAINS 4 
recently by Pascarella and Terenzini (1991) suggest that student learning or value-added in 
general education may follow significantly different patterns in different types of institutions. 

Research on the development of students' general cognitive skills (Pace, 1984, 1990; 
Winter, McClelland, & Stewart, 1981) suggested that students in selective liberal arts 
colleges experience significantly greater freshman to senior gains than do students attending 
other types of institutions, even when controlling for entering abilities. The greatest student 
gains in critical thinking were found in a study by Dressel and Mayhew (1954) at institutions 
with specially designated general education courses and core curricula requirements. 
Forrest (1982) studied general education gains at forty-four institutions by analyzing ACT 
COMP estimated freshmen to senior gains and the relationship between gains and 
institutional type. Again, he found that general education gains were highest at institutions 
devoting relatively large portions of their curricular requirements to general education. 

Pascarella and Terenzini (1991, p. 157) concluded that institutional characteristics 
appear to have some influence on the development of student general cognitive skills, but 
caution that "more methodologically sound studies, using objective measures and/or 
controlling for important precollege characteristics" are necessary. Thus an application of 
a structural equation modeling methodology to the study of freshmen to senior general 
education gains at institutions with varying approaches toward and student requirements for 
general education should be enlightening. 

An Overview of Structural Equation Modeling 

Structural equation modeling, otherwise known as latent variable modeling, has 
become a popular method for stu- lying student change and development over recent years 
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(Ecob, 1987). Structural equation models are designed to analyze changes in observed 
variables using unmeasured theoretical constructs, and they are based upon the assumption 
that relationships among observed variables are the products of relationships among latent 
constructs (James, Mulaik, & Brett, 1982: Long, 1983). Unlike the analysis of gain scores, 
which tends to produce nonsensical and counter-intuitive results, structural equation models 
are found to produce results consistent with theory and research (Hendrickson & Jones, 
1987). 

The power in structural equation modeling lies in the ability of the researcher to 
create hierarchies of models by successively freeing or constraining parameters (Pike, 1991). 
The hierarchies allow the comparison of theoretically acceptable models so that the model 
which gives the best explanation of the observed data can be determined (Joreskog & 
Sorbom, 1976). The fact that the same construct is being measured by pre- and post-tests 
in a prerequisite for change research. Structural equation modeling allows the stability or 
invariance of latent variables to be tested over time. The one requirement of the 
measurement model for assuming that latent variables represent true scores is that the 
observed variables must be congeneric, meaning they measure the same construct and are 
linearly related (Alwin & Jackson, 1981; Joreskog, 1971). 

A limitation of structural models is that they provide information exclusively about 
group effects (Muthen, 1989). Some relatively simple modifications have been suggested 
(Muthen, 1989) which change structural equation models from fixed-effect models 
appropriate for analyzing group differences to mixed-effect models appropriate for studying 
interactions between groups and individuals. Estimating latent variable means using a 
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FRESHMEN TO SENIOR GAINS 6 
mixed-effect model with multiple groups allows for the identification of effects specific to 
individuals and effects representing a conditional relationship or interaction between groups 
and individuals. 

Procedures 

The procedures paralleled those used by Pike (1991), with two differences: 1) ACT 
Assessment composite scores, rather than English and mathematics sub-scores, were used 
as the observed measures with the Pre-College Ability latent variable, and 2) groups were 
based on data from students attending institutions with different general education 
requirements as opposed to students with different course-taking patterns from the same 
institution, ACT Assessment composite scores were used because they were the only 
manifest variable associated with Pre-College Ability available in the data set received from 
ACT, Observed measures of both Freshmen Achievement and Senior Achievement were 
the three COMP Objective Test content sub-test scores: Functioning Within Social 
Institutions (FSI), Using Science and Technology (US), and Using the Arts (UA), The 
research model for the present study is shown in Figure 1, 
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Figure 1 The Research Model 
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FRESHMEN TO SENIOR GAINS 8 
The models were suggested by Pike (personal communication, February 18 and 24, 
1992) following the guidelines of Marsh and Grayson (1990). They assumed Freshmen 
Achievement was influenced by Pre-College Ability and Senior Achievement was influenced 
by both Pre-College Ability and Freshmen Achievement. ACT Assessment and COMP 
content scores were found to be normally distributed in preliminary analyses. Lambda 
parameters were fixed at unity for ACT composite and COMP Functioning Within Social 
Institutions scores. Effects parameters for Pre-College Ability were fixed at unity. The 
effects parameter for Freshmen Achievement was free to vary and the structural disturbance 
for Senior Achievement was set at zero. 

In each of the sets of analyses, hierarchical structural equation models with various 
degrees of constraints were compared against a baseline model using Tucker Lewis Indices 
(and Type-2 Relative Normed Fit Indices for the Mixed-Effect Models) in order to assess 
the goodness-of-fit for each model. Given that more than one model in each set met a 
minimum goodness-of-fit criterion (.90 for TLI and RNFI 2 ), the most parsimonious model 
was then selected as best. 

Three sets of analyses paralleling those of Pike (1991) were performed using LJSREL 
VII (Joreskog & Sorbom, 1989) based upon procedures suggested by Hayduk (1987). The 
first set of analyses examined the invariance of the measurement model across time 
(freshmen and senior scores) and the second examined the invariance of the measurement 
model across groups. The third set of analyses tested the observed data against several 
mixed-effects models (where more than one grouping factor is involved). Due to the large 
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sample size, a Tucker-Lewis Index was used in addition to the traditional chi-squared test 
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FRESHMEN TO SENIOR GAINS 9 
and a Type-2 Relative Normed Fit Index was also used for the mixed-effect models. 

The final procedures involved computing latent variable means using parameter 
estimates from the chosen mixed-effect model and structural equations [1] and [2] at given 
Pre-College Ability levels for the four groups and then testing for significant differences 
among the four groups in Freshmen to Senior Achievement gains. The structural equations 
used to compute latent variable means are shown below: 

E(^) - «i + Yn E(S X ) [1] 
E(? 2 ) « a 2 + p 2l E(i7x) + Yn E($ x ) [2] 

Invariance Across Time 

The first set of analyses used scores combined from the four groups based on general 
education programs to test the invariance of the measurement model over time. The first 
model constrained six measured variables which were the six COMP content sub-test scores 
to be uncorrected; this served as a standard against which other models could be evaluated 
(Marsh, Balla, and McDonald, 1988). The second model constrained Freshmen 
Achievement and Senior Achievement latent variables to be perfectly correlated. The third 
model was the same as the second except that the requirement of perfect correlation was 
relaxed- The fourth model kept the perfect correlation between latent variables, but allowed 
error terms to be correlated across times, 
Invariance Across Groups 

The second set of analyses involved evaluating the invariance of the measurement 
model across groups. The first model constrained six measured variables (freshmen and 
senior COMP content scores) to be uncorrected and provided a standard for evaluating the 
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other models. Model Two included the two latent parameters with lambda and epsilon 
coefficients allowed to vary across groups. The third model was similar to second except 
that lambda parameters were constrained to be equal across groups. Model Four had both 
lambda and epsilon parameters constrained to be the same across groups. 
Testing of Mixed-Effect Models 

The third set of analyses involved testing the data against several mixed-effect models 
proposed by Pike (personal communications, February 18 and 24, 1992). Model One was 
comprised of seven uncorrected measured variables; it was the same as the standard model 
used earlier to test for invariance of measurement across groups except that ACT scores 
were also included. The second model was a mixed-effect model whereby students in the 
four groups with Pre-College Ability scores constrained to be equal had different Freshmen 
and Senior Achievement scores. Model Three constrained the parameters in the structural 
equation for Freshmen Achievement to be the same for all groups, implying that students 
with the same pre-College Ability scores would have the sair t Freshmen Achievement 
scores, but different Senior Achievement scores. The fourth model constrained the 
parameters in the structural equations for both Freshmen and Senior Achievement to be the 
same for the four groups. This meant students with the same Pre-College Ability scores 
would have the same Freshmen and Senior Achievement scores. A fifth model, where all 
parameters were constrained to be invariant across groups was used to serve as a baseline 
for computing Type-2 Relative Nonned Fit Indices for the third and fourth models. 
Testing for Significant Differences in Latent Freshmen to Senior Achievement Gains Among 
the Four Groups 
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Maximum likelihood estimates for the selected mixed-effect model were used along 
with equations [1] and [2] to compute latent freshmen and senior achievement means for 
the four groups. Differences in achievement gains for the four groups were tested for 
significance using a latent variable multivariate analysis of variance (MANOVA) procedure 
suggested by Bagozzi and Yi (1991). Planned comparisons of latent achievement gains 
were used to detect where significant differences occurred among the four groups. 

Sample 

The study sample consisted of 6,409 students from thirty-four institutions for whom 
ACT Assessment, and freshmen and senior ACT COMP Objective Test scores were 
available in a data set compiled by ACT. Sixty-three percent of students in the data set 
attended public institutions. Thirty-two percent of institutions in the data set were research 
universities as classified by Carnegie type; nine percent were doctoral-granting universities; 
thirty-two percent were comprehensive institutions; and twenty-seven percent were liberal 
arts colleges. 

Fifty-nine percent of students in the data set were female and students' average age 
at the time of taking the COMP as seniors was 21.5. Their mean ACT Assessment 
composite score was 21.1. Mean freshmen total COMP Objective Test score was 176.0. 
Mean senior COMP Objective Test score was 184.6, representing a mean score gain of 8.6 
for this group. 

A final variable in the ACT data set corresponded to type of general education 
program in place at the institution students attended when taking the Objective Test as 
freshmen and again as seniors. Institutions were grouped according to Forrest's (1982) 
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classification scheme based upon general education curricular characteristics. The four 
groups included 1) programs where general education distribution requirements comprised 
forty percent or more of the curriculum and were evenly divided between courses in written 
and oral communication, social sciences and history, natural sciences and mathematics, and 
humanities and fine arts (five institutions, N = 218), 2) programs with thirty-nine percent 
or less of the curriculum devoted to general education and no equal distribution (four 
institutions, N = 2,319), 3) programs with either forty percent or more of the curriculum 
devoted to general education or equal distribution requirements, but not both (twenty-three 
institutions, N = 3,281), and 4) programs with core general education curricula (two 
institutions, N = 591). 

Results 

Invariance Across Time 

The results of hierarchical models used to test for invariance across time are shown 
in Table 1. While all of the models had significant chi squares, this was not taken 
conclusively to mean that the models did not represent the data due to the veiy large 
sample size. Models 2, 3, and 4 all provided an acceptable fit to the data because they had 
Tucker-Lewis Index values greater than .90 using the procedures recommended by Marsh, 
Balla, and McDonald (1988). Having established that all three models provided an 
acceptable fit to the data, Model 2, the time invariance model, which suggested the ACT 
COMP was a congeneric test which measured the same underlying construct for freshmen 
and seniors in the sample, was employed throughout subsequent analyses since it 
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represented the most parsimonious model 

Table 1 Goodness -of -Fit Results for Models of Invariance Over Time 



MODEL 


df 


x 2 


TLI 


Baseline Model 


15 


12 , 212 • 66* 




Time Invariance 


10 


83-11* 


0.99 


Common Factors 


10 


83.11* 


0.99 


Correlated Errors 


7 


36.09* 


0.99 



* p < .001 

Invariance Across Groups 

Results of the use of hierarchical models to test for invariance across groups are 
shown in Table 2. The second, third, and fourth models were all found to provide adequate 
representations of the observed data since they showed Tucker-Lewis indices in excess of 
.90. The upsilon or pattern invariance model [2], suggesting the same constructs were being 
measured across groups, was employed for the final set of analyses because it represented 
the most parsimonious model. 

Table 2 Goodness-of -Fit Results for Models of Invariance Across Groups 

MODEL df X 2 TLI 

Baseline Model 60 11,755.81* 

Pattern Invariance 40 132.61* 0.99 

Lambda Invariance 46 197.04* 0.98 

Error Invariance 64 317.35* 0.98 



* p < .001 
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Testing of Mixed-Effect Models 

The final set of LISREL analyses was used to distinguish between hierarchical mixed- 
effect models* Model 2 allowed students in the four groups who had the same Pre-College 
Ability scores to have different Freshmen and Senior Achievement scores. Model 3 held 
Freshmen Achievement scores constant for students with identical Pre-College Ability 
scores, but allowed them to have different Senior Ability scores. Model 4 held both 
Freshmen and Senior Achievement scores constant for students with the same Pre-College 
Ability scores. 

The results are shown in Table 3, All three models had Tucker-Lewis indices above 
,90, Neither Model 3 nor Model 4 had Type-2 Relative Normed Fit Indices at or above ,90, 
so they were not selected as superior to Model 2, The most parsimonious model [2], which 
allowed Freshmen and Senior Achievement to vary for students with the same levels of Pre- 
College Ability, was used with equations [1] and [2] to calculate scores for Freshmen and 
Senior Achievement given various pre-college ability scores. The maximum likelihood 
estimates for this model are shown in Table 4, Latent variable means for Freshmen and 
Senior Achievement at various levels of Pre-College Ability are shown in Table 5, 
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Table 3 Goodness-of-Fit Results for Mixed-Effect Models 
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Table 4 Selected Maximum Likelihood Estimates for the Mixed-Effect 
Model 



PARAMETER 


GROUP 1 


GROUP 2 


GROUP 3 


GROUP 4 


Alpha,] 


39.099 


37.559 


38.271 


37.513 


Gamma^ 1 


1 


1 


1 
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Alpha 2 


45.735 


44.618 


46.199 


43.028 


Gamma 2 ^ 


1 


1 


1 


1 


Beta 2J 


-0.075 


-0.066 


-0.086 


-0.044 
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Table 5 Latent Variable Meana for Freshmen and Senior Achievement at Pre-College 
Ability Levels of 10, 20, and 30 

PRE -COLLEGE ABILITY = 10 



LATENT 
VARIABLE 

MEAN GROUP 1 GROUP 2 GROUP 3 GROUP 4 



Freshmen 

Achievement 49.099 47.559 48.271 47.513 

Senior 

Achievement 52.053 51.479 52.048 50.937 

PRE-COLLEGE ABILITY = 20 



LATENT 
VARIABLE 

MEAN GROUP 1 GROUP 2 GROUP 3 GROUP 4 



Freshmen 

Achievement 59.099 57.559 58.271 57.513 

Senior 

Achievement 61.303 60.819 61*188 60*497 

PRE-COLLEGE ABILITY = 30 



LATENT 
VARIABLE 

MEAN GROUP 1 GROUP 2 GROUP 3 GROUP 4 



Freshmen 

Achievement 69.099 67.559 68.271 67.513 
Senior 

Achievement 70.553 70.159 70.328 70.057 



Testing for Significant Differences in Freshmen to 

Senior Latent Achievement Gains Among the Four Groups 

A multivariate analysis of variance (MANOVA) procedure suggested by Bagozzi and 
Yi (1991) was used to test for significant differences among freshmen to senior latent 
difference scores for the four groups by comparing the chi square and degrees of freedom 
resulting from the chosen mixed-effect model to that of a second model where latent 
variable means were constrained to be equal across groups. The results are shown in Table 
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6, The results indicated a significant difference occurred between the two models, indicating 
evidence to reject the null hypothesis that latent achievement gain scores were equal across 
groups. 

Table 6 Latent Variable MANOVA Results Used to Test the Null Hypothesis of No 
Difference in Achievement Gains Among the Four Groups 

MODEL df x 2 Adf Ax 2 

Mixed-Effect 

Model 82 426 ♦ 60 

Null Model 
(Latent Means 
Equal Across 

Groups) 88 497,22 6 70.62* 

p < ,01 

Finally, planned comparisons were carried out to determine which groups differed 
in gain scores among latent achievement levels. To accomplish this, the results of the 
chosen mixed-effect model were compared against the results of two models where latent 
variable means were constrained to be equal across 1) group one and group two, and 2) 
group two and group four. The results, shown in Table 7, showed there were significant 
differences in achievement gains between groups one and two, but not between groups two 
and four. Gains were significantly greater for students in group two institutions than for 
students in group one institutions. 
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Table 7 Results of Planned Comparisons Between Groups One and Two 
and Groups Two and Four 

MODEL df X 2 Adf A X 2 

Mixed-Effect 

Model 82 426.60 

Group One and * 
Two Means Equal 88 497.22 6 70.62 

Group Two and 

Four Means Equal 84 426.97 2 0.37 

* p < .01 

Discussion 

The results showed that structural equation modeling represented a useful method 
of representing the interactions between student characteristics and general education 
program effects. The theoretical constructs of Pre-College Ability, Freshmen Achievement, 
and Senior Achievement and their interrelationships first noted by Pike (1991) were found 
to adequately represent the data collected from a nation-wide group of colleges and 
universities. The important abilities of structural equation models to demonstrate invariance 
over time and across groups proved useful in this study. 

The best (most parsimonious) model which adequately represented the data was one 
where both Freshmen and Senior Achievement varied with given fixed levels of Pre-College 
Ability. This is an interesting finding because it says that given the same ACT Assessment 
composite scores, freshmen students' COMP scores are not the same across institutions. 
This speaks well for the COMP's ability to detect different general education knowledge and 
skill levels given the same ACT composite scores, but leads to the question of why students 
with the same Pre-College Ability levels show different levels of freshmen general education 
achievement across institutions in the four groups. Perhaps there is some self-selection of 

13 
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students into particular types of institutions even given the same levels of high school ability. 

The study showed that students in group two institutions (where less than forty 
percent of the minimum undergraduate curricular requirements were devoted to general 
education courses and where there was not equal distribution of the curriculum among the 
general education requirement) showed significantly greater COMP gains than students in 
group one institutions (where forty percent or more of undergraduate students' requirements 
were in general education and there was equal distribution of communication, mathematics, 
social sciences, natural sciences, and humanities courses within the general education 
requirement). 

This finding is exactly the opposite of that of Forrest (1982), whose study of the 
effects of institutional general education requirements on students' COMP score gain 
showed an average COMP total score gain of 8.9 score points for students from institutions 
with the same characteristics as the present study's group one and an average gain of 3.8 
points for students from institutions with the same general education requirements as the 
present study's group two. 

Three possible reasons are suggested for the contradictory results of Forrest's (1982) 
study and the present research. The provision of error free measurement afforded by the 
methodology of the present study overcame a limitation of Forrest's (1982) effort in which 
the results were obscured due the problems in the study of gain scores. A second 
explanation for the two different research results may be that the grouping of institutions 
in the present study may not be optimal. The third explanation offered for differences in 
the results of the two studies is that intervening demographic and educational characteristics 
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may have influenced the present research. Several variables were compared between 
institutions and students included in groups one and two and considerable differences 
existed between institutional type, control, and enrollment, and somewhat of a difference 
in student gender, between group one and two institutions. 

Curricular implications resulting from the present study are that the definitions, 
course content, and instruction pertaining to general education maybe considerably different 
between institutions in the sample and that which is expressed in the COMP Objective Test 
Policy implications include the idea that a relatively large percentage of the undergraduate 
curricula devoted to general education and equal distribution of course content within the 
general education requirement are not necessarily positively related to greater gains in 
student achievement (at least as measured by the COMP). This suggests changes in student 
requirements or at least a more critical study of the effects of general education programs. 

One limitation was that there was some subjectivity in the grouping of the institutions 
(i.e. what constitutes equal distribution of courses in some cases). Another limitation may 
have been that the timing of the pre-testing at each institution varied and that more than 
one freshmen to senior cohort may be included for each institution. 

Other useful applications of this methodology to research in general education gains 
might include examining the effects of some of the demographic and institutional variables 
noted above, in addition to institutional general education requirements, on general 
education gains and using of the same methodology with students at institutions where 
general education assessments other than the COMP are utilized. 
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Abstract 

• * A-ef-mce* in freshmen to senior student general 
This study sought to investigate differences in tresnm 

• • „™no natterns of general education requirements 

education gains across institutions with varying patterns g 

using a mixed-effect structural equation model. The subjects were 6,409 students at thirty- 
four nation-wide colleges and universities. Students attending institutions where less than 
forty percent of undergraduate curricular requirements were devoted to general education 
^ where there was not equal distribution of general education courses within the 
re quirement were found to have significantly higher general education gains than did 
stud ents who attended institutions where forty percent or more of the undergraduate 
— was devoted to general education and there was equal distribution of courses 
within the general education requirement. 
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